Econometrics at Scale: Spark up Big Data in Economics

This paper provides an overview of how to use “big data” for social science research (with emphasis on economics and finance). We investigate the performance ease different Spark applications running a distributed file system enable handling analysis data sets which were previously not usable due their size. More specifically, we explain (i) explore big exceed retail grade computers memory size (ii) run typical statistical/econometric tasks including cross sectional, panel time series regression models are prohibitively expensive evaluate stand-alone machines. By bridging gap between abstract concept ready-to-use examples can easily be altered suite researchers need, provide economists scientists more generally with theory practice handle ever growing datasets available. The reproducing in this makes guide useful reference limited background computing.